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The purpose of this study was to utilize thermal imaging and the Concealed Information 
Test to detect deception in participants who committed a mock crime. A functional 
analysis using a functional ANOVA and a functional discriminant analysis was conducted 
to decrease the variation in the physiological data collected through the thermal imaging 
camera. Participants chose between a non-crime mission (Innocent Condition: IC), or a 
mock crime (Guilty Condition: GC) of stealing a wallet in a computer lab. Temperature in 
the periorbital region of the face was measured while questioning participants regarding 
mock crime details. Results revealed that the GC showed significantly higher temperatures 
when responding to crime relevant items compared to irrelevant items, while the IC did 
not. The functional ANOVA supported the initial results that facial temperatures of the GC 
elevated when responding to crime relevant items, demonstrating an interaction between 
group (guilty/innocent) and relevance (relevant/irrelevant). The functional discriminant 
analysis revealed that answering crime relevant items can be used to discriminate guilty 
from innocent participants. These results suggest that measuring facial temperatures 
in the periorbital region while conducting the Concealed Information Test is able to 
differentiate the GC from the IC. 
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INTRODUCTION 

Deception detection is widely used by law enforcement around 
the world. Although very few countries actually allow the results 
to be used as evidence in court, investigators frequently use lie 
detecting as a tool of reference during investigations. Many forms 
of deception detection exist, but the polygraph is the most widely 
used method. Unfortunately, field studies have shown that poly- 
graph testing accuracy is in the unsatisfactory range of 72-91% 
(National Research Council, 2003). Among the numerous rea- 
sons for the variability in accuracy, a main drawback of polygraph 
testing is its dependency on the level of training and experience 
of the polygrapher. In other words, the accuracy of a polygraph 
test is greatly affected by the subjective skill of the polygrapher. 
Also, polygraph testing in itself can cause high levels of anxiety in 
subjects, which can also affect the results or even lead to false- 
positive conclusions. It is therefore imperative that additional 
means of deception detection are developed, standardized, and 
applied as alternative methods, or at least as secondary support to 
the polygraph. 

Deception detection using thermal imaging (a.k.a. thermogra- 
phy) incorporates an infrared thermal imaging camera to mea- 
sure facial skin temperature as a cue to deception. Although not 
yet used in law enforcement, thermal image analysis for polygraph 
testing has already gained a US patent (Pavlidis, 2005; Patent 
No: US 6854879 B2), and has obtained empirical support from 



previous research with results suggesting that it has the potential 
to detect deception quite accurately (Pavlidis et al., 2002; Pollina 
et al, 2006; Tsiamyrtzis et al, 2007; Dowdall et al, 2009). In 
general, when a deceptive subject is being interrogated, they expe- 
rience stress which activates the autonomic nervous system. This 
then activates the sympathetic nervous system, which is respon- 
sible for stress responses such as increased blood flow to the eyes 
to facilitate rapid eye movement in preparing the body for the 
fight-or- flight response (Pavlidis and Levine, 2001, 2002). This 
increased blood flow is detectable in the periorbital region of 
the face through thermal imaging. The periorbital regions are 
the symmetrical areas to the left and right of the bridge of the 
nose between the eyes. Previous deception detection studies that 
used thermal imaging also did so by measuring the temperature 
of the periorbital regions of the face. In these studies, average 
facial temperatures collected from the periorbital regions were 
higher during deceptive responses, compared to non-deceptive 
responses, thus acting as cues to deception. 

An outstanding advantage of using thermal imaging is that it is 
non-invasive, in that no sensors are attached to the subject (Arora 
et al., 2008). While typical polygraphs require numerous con- 
tact sensors, thermal imaging has none, making it more natural 
and comfortable. Research in psychophysiology has shown that 
contact sensors (i.e., polygraph sensors) can compromise com- 
fort, which can effect physiological measurement (Yankee, 1965), 
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as well as deception detection procedures (Pavlidis and Levine, 
2002). Another advantage is that raw thermal data can be saved 
for later analysis, in the case that a better more accurate analy- 
sis method is developed in the future (Pavlidis and Levine, 2001). 
In addition, thermal imaging cameras generally look like video 
cameras, meaning deception detection could take place without 
the subject even realizing it is happening, which can prevent 
unwanted attempts at countermeasures. 

While detecting deception with thermal imaging has many 
advantages, it also has certain disadvantages, one of them being 
that it is sensitive to the environment and changes in the envi- 
ronment. In particular, it is sensitive to ambient temperatures 
and humidity levels (Hermans-Killam, 2002), as well as changes 
in the distance between the subject and the thermal imaging 
camera lens (Jones and Plassmann, 2002). Unlike measuring 
body temperatures to detect sick people at an airport, detecting 
deception must measure very small changes in skin surface tem- 
perature, and therefore such sensitivity may have a critical effect 
on the measurement results. Therefore, in order to control for 
these possible variables, the present study conducted the ther- 
mal imaging measurements in a highly controlled experimental 
environment. 

To detect deception, whether using a polygraph or ther- 
mal imaging, a method of questioning is needed. Although in 
the field the Control Question Test (CQT; Reid, 1947) is the 
most widely used questioning technique (Meijer and Verschuere, 
2010), it is criticized by researchers for its lack of theoretically 
based empirical evidence (Ben-Shakhar, 2008; Iacono, 2008). 
Unlike the CQT, the Concealed Information Test (CIT; a.k.a. 
Guilty Knowledge Test or GKT; Lykken, 1959) is empirically 
supported as a physiologically sound method of questioning 
(Ben-Shakhar and Furedy, 1990; Elaad, 1998; MacLaren, 2001; 
Ben-Shakhar and Elaad, 2003). The present study detected decep- 
tion under controlled experimental conditions, and therefore 
utilized the CIT instead of the CQT to maintain a theoretically 
based experimental process of deception detection. In addi- 
tion, because the second experimenter (the interviewer) was 
not trained in interrogation, the CIT is ideal in that it is a 
standardized, easily replicated procedure that does not require 
professional training, as does the CQT (Ben-Shakhar and Elaad, 
2002). 

Unlike field studies where the interviewees are suspects to 
actual crimes, participants in experiments are typically aver- 
age citizens or students, and therefore a mock crime is needed. 
Guilty participants commit a crime, and innocent participants 
enact a similar non-criminal task, or are simply given infor- 
mation about the crime. To motivate participants and provide 
an incentive to be judged innocent, they are given a reward 
(e.g., monetary compensation, academic credits) upon success- 
ful deception, or punishment (e.g., monetary penalties, academic 
tasks) for failing to deceive. In a meta-analytic study of mock 
crime research, the incentive to motivate deception was a main 
variable that affected the outcome of deception detection (Kircher 
et al, 1988). Therefore, in the present study, participants in the 
guilty condition were told they would receive triple the original 
participation fee upon success, but would receive nothing if they 
failed, incorporating both award and punishment. 



To further increase anxiety during the mock crime, guilty par- 
ticipants were to commit theft and eliminate evidence of their 
crime in a public computer lab. Innocent participants had to go 
to the same computer lab and send out an email, which allowed 
the blind experimenter to ask questions that were relevant to both 
groups, but only the guilty participants would possess crime- 
relevant information. Further details regarding the mock crime 
scenario are explained in the method section. 

As with most physiological data, skin surface temperatures 
measured using thermal imaging could be thought of as func- 
tional data, and was therefore further analyzed using a functional 
ANOVA (Ramsay and Silverman, 2006) and a functional dis- 
criminant analysis (Ramsay and Silverman, 2006). An important 
property that distinguishes functional data from multivariate data 
is the existence of a smooth curve assumed to generate the data. 
Functional data assumes that an underlying function gives rise to 
the observed data, and that the underlying function is smooth 
so that adjacent data values tend to be similar to some extent 
and not too different from each other. In other words, adjacent 
data values provide overlapping information, not independent 
information. 

A functional ANOVA is a functional extension of an ANOVA, 
in which the response variable is a function and predictor vari- 
ables are categorical. A functional ANOVA was used to see if facial 
temperatures of participants were affected by guilt or innocence 
and/or whether they were answering crime-relevant or crime- 
irrelevant questions. The functional discriminant analysis is a 
functional version of Fisher's linear discriminant analysis, which 
seeks to find components, or weighted integrations of functions 
that separate multiple groups of observations as much as possi- 
ble. The functional discriminant analysis was used to see how well 
the facial temperature data was able to differentiate guilty partic- 
ipants from the innocent participants. Further details regarding 
the functional ANOVA and the functional discriminant analy- 
sis are explained in the method section and the appendix of this 
study. 

The aim of the present study was to detect deception in par- 
ticipants who conducted a realistic mock crime using infrared 
thermal imaging and a simplified facial tracking method, along 
with the CIT method of questioning. The purpose of the study 
was to (a) detect deception using thermal imaging through a sim- 
plified method, (b) in a more controlled environment, (c) using 
the most realistic mock crime possible, (d) using the most opti- 
mal method of statistical analyses, and (e) replicate the results of 
the previous studies that have done so in the past. It was pre- 
dicted that the guilty participants would be differentiable from 
the innocent participants, in that the guilty condition would 
show an increase in facial temperatures of the periorbital regions 
when responding to crime-relevant sub-questions compared to 
the irrelevant sub-questions of the CIT, while the innocent con- 
dition (IC) would show no significant difference between the 
two. It was also predicted that using the same thermal imaging 
data, a functional ANOVA would reveal similar results, support- 
ing the initial analysis, and also that a functional discriminant 
analysis would be able to differentiate the guilty from the inno- 
cent participants from their facial temperatures in the periorbital 
regions. 
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METHOD 
PARTICIPANTS 

A total of 34 participants were recruited from an online bulletin 
board on a university website. The bulletin board entry stated 
that participants were being recruited for a psychology experi- 
ment on measuring facial temperatures using thermal imaging, 
and would be paid $10 for their participation. All participants 
read and signed a written consent form agreeing to participate in 
the experiment. One participant was unable to finish the exper- 
imental procedure, and the thermal imaging data from three 
participants was incomplete and had to be discarded. This left 
the data of 30 participants (17 male, 13 female), between the 
ages of 18 and 30 (M = 22.74, SD = 2.77), for the final data 
analyses. 

MATERIALS 
Apparatus 

Thermal imaging. To record the facial temperatures of the par- 
ticipant's faces during the experiment, an Infrared Thermography 
H2640 infrared thermal imaging camera (NEC Avio Infrared 
Technologies Co. Ltd., Japan) with 320 x 240 pixel resolution 
and heat resolution of 0.08°C (±2% accuracy) at 30 Hz mounted 
on an industrial strength tripod (SLIK Corporation, Japan) was 
used. The thermal imaging camera was placed so that the lens 
of the camera was 100 cm (±1 cm) from the participant's face, 
which is the distance that the thermal imaging camera manufac- 
turer suggested as the optimal recording distance for measuring 
human skin surface temperatures. The thermal imaging cam- 
era was connected to an Xnote P300-TP8WK laptop computer 
(LG Electronics, Korea). A digital thermometer/hygrometer was 
placed directly under the thermal imaging camera, and experi- 
ments were conducted at a constant room temperature of 21.0°C 
(±0.25°C) and 65% (±2%) humidity. 

Webcam. To provide a CCTV security camera at the computer 
lab where the mock crime would be taking place, a Quick Cam® 
Ultra Vision SE webcam (Logitech, USA) was mounted at the 
front of the computer lab. The webcam was connected to a 
desktop computer at the desk where a confederate acting as the 
computer lab assistant was sitting. This webcam not only acted as 
a CCTV security camera which the participants conducting the 
mock crime had to deactivate, it also allowed the experimenters 
in the psychology laboratory to view what was happening in the 
computer lab while the mock crime was taking place. 

The red wallet. A bright-red, faux leather, woman's wallet with 
gold-plated trimming was used as the target object that the par- 
ticipants conducting the mock crime had to steal. The wallet was 
a three-way folding style wallet with a few credit cards, some busi- 
ness cards, and some monetary bills placed in it to make it look 
and feel as realistic as possible. 

Health questionnaire 

A short questionnaire was designed to ask participants whether 
they were sick, taking any kind of medication, had any history 
of thyroid problems which may affect body temperature con- 
trol, or were currently visiting the hospital for any of the above 



reasons. This questionnaire was conducted before the experi- 
ment to screen out any possible participants who may not show 
"normal" physiological or temperature related responses to the 
experimental procedures. 

Concealed Information Test 

While recording the thermal imaging data, participants were 
asked a series of questions to detect deception. Each question 
begins with a main primary question, followed by a series of five 
secondary sub-questions containing different possible answers to 
the original main question. For example, a main question was 
"What was the item you stole from the computer lab?" and was 
followed by sub-questions such as "Was it a watch?" "Was it a 
ring?" and "Was it a wallet?" According to the theories underly- 
ing the CIT, if the participant did actually steal the wallet, then 
he or she would have critical knowledge regarding the mock 
crime, which in this case would be the wallet. Thus, when a guilty 
participant is asked if the item stolen was a wallet, their facial tem- 
perature response to this sub-question would be different from 
the other sub-questions presented. Three main questions were 
asked, but there were only two actual questions. The first question 
was asked a second time after the second question to conform to 
the standard practice of the CIT, with the sub-questions being in a 
different order from the first time to eliminate any ordering effects 
(see Appendix section "Concealed Information Test Questioning 
Protocol" for the questioning protocol). The sub-questions that 
were not relevant to the crime are irrelevant items (IR), and the 
crime relevant sub-questions are relevant items (RE). After each 
sub-question was asked, a period of 10 s was allowed to pass 
before the next question was asked. This was to allow the partic- 
ipants' facial temperatures to recover from any fluctuations that 
may have occurred from the previous question. The questioning 
session lasted approximately 6-8 min. 

PROCEDURE 

Upon arrival, participants first read and signed a written consent 
form, followed by the health questionnaire. The experimenter 
then entered the room carrying two colored envelopes and some 
yellow Post-it notes. The experimenter explained the experiment 
and what the participant would have to do (see Appendix sec- 
tion "Envelope Selection Explanation"). The participants then 
chose whether they would conduct the guilty (red) or innocent 
(blue) mission. The experiment was purposefully designed this 
way to increase the level of involvement the participants would 
feel. Considering most crimes are committed by choice, and not 
by force, it was expected that this method would produce a more 
realistic response during deception detection. Participants were 
informed that if they successfully accomplished the red mission, 
they would receive a reward three times the amount of the original 
$10 participation fee of $30, but if they failed they would receive 
nothing. Here, success meant stealing the wallet and eliminating 
the evidence without being discovered or questioned by anyone, 
and not being detected of deception by the second experimenter 
(interrogator). They were also informed that if they successfully 
accomplished the blue mission, they would receive a $10 partici- 
pation fee. Here, success meant sending an email at the computer 
lab, leaving a message to the computer lab assistant, and not 
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being accused of deception by the second experimenter. Although 
the participants were informed that failing the missions would 
result in receiving nothing, all participants were to receive a small 
token of appreciation of $5 for participating in the experiment, 
regardless of the outcome. 

The participants selected an envelope, read the instruction 
sheet inside, and were also given a small Post-it note to write down 
whatever information they thought was important in completing 
the mission. The Post-it notes were small and did not have enough 
space to copy all the mission information verbatim. This forced 
participants to summarize the information in their own words. 
It was expected that this process would further increase personal 
involvement in the mock crime, leading to a feeling of having 
planned certain aspects of the mission themselves, or a feeling 
of having taken part in the plotting of the crime. The mission 
documents contained the following instructions: 

Blue mission: 

1. Go to the Computer Lab located on the 5th floor of this 
building. 

2. Sit at any seat and send an email to kekipark@hotmail.com. 

3. In the body of the email, write your name, age, and mobile 
phone number. 

4. When the Computer Lab Assistant leaves his seat, go to his 
computer and leave a note using the Notepad program on his 
computer saying "Hello, I used one of the computers. Thank 
you." 

5. Return to the Psychology Laboratory. 
Red mission: 

1. Go to the Computer Lab located on the 5th floor of this 
building. 

2. Locate the girl sitting in the 3rd row from the front of the 
room, and the 4th seat from the windows. 

3. When the girl leaves her seat, go to her seat and steal a red 
wallet from her bag. 

4. When the Computer Lab Assistant leaves his seat, go to his 
computer and shutdown the webcam program that is running 
on the desktop. This will automatically delete the video file 
being saved. 

5. Return to the Psychology Laboratory. 

After participants finished writing their notes, they were given 
detailed explanations on how to get to the computer lab. They 
were then told to go the computer lab, execute the mission, and 
immediately return as soon as they were done. 

Upon returning, participants were asked if they had success- 
fully completed their mission, and guilty participants were asked 
for the wallet. The participants were then taken to a temper- 
ature and humidity controlled measurement room where the 
thermal imaging camera was set up. A second experimenter blind 
to the participants' mission selection informed the participants 
that although she was aware that a crime had taken place in the 
computer lab, she had no knowledge of who the perpetrator was. 
She then explained that she would ask a series of questions in 
an attempt to figure out whether the participant committed the 
crime or not. 



Before questioning, participants relaxed for 2 min to adjust to 
the room. Afterwards, the first experimenter came into the room 
to adjust the thermal imaging camera to record a 1 min base- 
line reading. The participants were told that the camera was a 
video camera, and that the interview would be recorded and later 
analyzed, so to remain as motionless as possible during ques- 
tioning, and to maintain eye-contact with the camera until the 
questioning ended. The second experimenter was seated facing 
the participant at a right angle and was outside the field of view 
of the participant. As questioning began, the first experimenter 
began recording the thermal data from outside the room with 
a laptop computer connected to the thermal imaging camera. 
The entire experiment lasted approximately 45 min to 1 h, includ- 
ing the questionnaires, explanation and task selection, the mock 
crime, and the questioning session. When finished, participants 
were thanked, debriefed, and asked not to disclose any informa- 
tion regarding the experiment until the end of the experiment 
period, to prevent contaminating future participants. 

EXPERIMENTAL DESIGN 

There were two experimental conditions in this study: 18 par- 
ticipants in the Guilty Condition (GC) which selected the red 
envelope and committed a mock crime, and 12 participants in the 
IC which selected the blue envelope and acted as the control con- 
dition. Therefore, in order to differentiate which participants were 
in the GC and which were in the IC, the average of the maximum 
temperature values in the periorbital region while responding to 
the RE questions were compared to the values while responding to 
the IR questions. Although the first primary question was asked 
twice, and the second primary question asked once, each repe- 
tition of the first primary question was treated as an individual 
primary question in the analysis. Therefore, there were a total 
of three primary questions in the analysis. The mean tempera- 
ture values of the RE and the IR items for each condition were 
compared using a paired-samples t-test. A significant increase in 
mean temperature value for the responses to the RE compared to 
the IR sub-questions would signal that the participant possessed 
concealed knowledge regarding the mock crime. 

DATA COLLECTION AND ANALYSES 
Initial analysis 

The thermal image data used to analyze the facial temperature 
readings were collected from the periorbital region of the face. 
This is the area between the eye and the bridge of the nose on 
either side of the nose. As shown in Figure 1, an area of inter- 
est (AOI) was designated to cover the periorbital regions, but 
not the actual eye itself. An AOI is an area designated by the 
user of the thermal imaging software from which maximum or 
minimum temperatures are collected and analyzed. AOIs are des- 
ignated in order to avoid including areas of the face which are 
always the hottest regions regardless of the situation, such as the 
eye sockets and the inside of the mouth. The maximum temper- 
ature point within the AOI was recorded during each frame of 
recording (30 frames per second). The mean temperature value 
corresponding to each response was the average of the maxi- 
mum temperature point during the 10 s of response time given 
after each sub-question was asked. The 10 s of response time 
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FIGURE 1 | A sample thermal image showing the Area of Interest (AOI) 
covering the periorbital regions (left and right corners of the inverted 
triangle) and the metallic tracking sticker (black dot). 



started at the end of the last word of each sub-question. When 
an AOI is selected, the thermal imaging software automatically 
tracks this designated region of the face and follows it when the 
participant moves their face. However, to increase tracking accu- 
racy, a small metallic sticker was placed above the bridge of the 
nose which in thermal imaging appears as a black dot relative 
to the skin (see Figure 1). Therefore, when the AOI was set to 
follow the black dot, tracking was extremely accurate as long as 
the participants did not tilt their head from side to side at an 
angle or turn their head to the left or right. None of the partici- 
pants tilted or turned their heads during measurement. The point 
of maximum temperature was always measured from within the 
designated AOI. 

To compare mean facial temperature values between condi- 
tions, independent-samples f-tests were conducted, and to com- 
pare between RE and IR sub-questions, paired-samples f-tests 
were conducted, all using SPSS 17.0 for Windows. 

Functional ANOVA and functional discriminant analysis 

Facial temperatures were measured for a duration of 10 s 
beginning after each question was posed by the experimenter. 
Therefore, the data consisted of 450 time series, or functions 
(30 participants x 15 questions), measured over 300 time points 
(10 s x 30 Hz), and three participants were eliminated from 
the analysis due to severe noise in their signals. Due to the 
limitation of computational power, the number of time points 
needed to be reduced to conduct the functional ANOVA, and 
therefore one of every five time points was used so that the 
number of time points per question was decreased to 60 (10 s x 
6 Hz). A total of 450 functions measured over 60 time points 
were analyzed. Figures 2A,B display the raw data of one guilty 
subject (subject 1) measured while answering three relevant 
questions and 12 irrelevant questions, respectively. Similarly, 
Figures 2C,D show the raw data of one innocent subject (sub- 
ject 3) measured while answering three relevant questions and 
12 irrelevant questions, respectively. Before any analyses were 



conducted, the original functions were smoothed by the rough- 
ness penalty smoothing method with X = 10 (see Appendix 
section "Smoothing: Roughness Penalty Smoothing Method" for 
more details on smoothing and Appendix section "Functional 
ANOVA" for details on the functional ANOVA). Figure 3 dis- 
plays the smoothed data corresponding to the raw data shown in 
Figure 2. 

The functional discriminant analysis estimates a weight func- 
tion, instead of a vector of weight, which separates multiple 
groups of functions as much as possible (see Appendix section 
"Fisher's Linear Discriminant Analysis" for the technical details 
of the Fisher's linear discriminant analysis and Appendix section 
"Functional Discriminant Analysis" for details on the functional 
discriminant analysis applied). The data used in the functional 
discriminant analysis consisted of 450 time series, or functions, 
(30 participants x 15 questions) measured over 300 time points 
(10 s x 30 Hz). The data measured for RE questions and IR 
questions was analyzed separately. 

RESULTS 

HEALTH QUESTIONNAIRE 

No participant reported any medical problems in the Health 
Questionnaire. 

BASELINE FACIAL TEMPERATURES 

An independent-samples f-test revealed no significant differ- 
ences in baseline facial temperature readings between the GC 
(M = 35.83, SD = 0.59), and the IC (M = 36.03, SD = 0.78), 
t(28) = —0.78, p = 0.44. There were also no significant differ- 
ences between male and female participants, or between their 
ages. These results show that there was no significant facial tem- 
perature difference between the participants in the GC and the IC 
before the experiment began. 

FACIAL TEMPERATURE CHANGE VALUES 

The thermal imaging camera measured temperatures at 30 frames 
per second. For each sub-question asked, temperature values of 
the hottest point within the AOI were recorded for a period of 
10 s starting at the moment the experimenter ended her ques- 
tion. These temperature values were averaged, resulting in a mean 
facial temperature value for the RE items and the IR items for 
each participant. To obtain a facial temperature change value 
(FTCV) for the RE and IR items, baseline facial temperatures was 
subtracted from the mean temperature values. 

Before performing paired-samples f-tests as described in the 
following sections, the normality assumption was tested which 
should be satisfied for a paired-samples f-test to be conducted. 
First, scatter plots, Q-Q plots, and boxplots of the FTCV scores 
of the four conditions (GC-RE, GC-IR, IC-RE, and IC-IR) were 
examined, and are presented in Figures 4, 5, and 6. As shown 
in Figures 5 and 6, the FTCV scores of subject 31 of the GC for 
both the RE and IR questions seemed to be deviated from normal 
distributions. Therefore, the Shapiro-Wilk test was performed to 
statistically test the null hypothesis that the FTCV scores in each 
of the four conditions came from a normal distribution. Results of 
the Shapiro-Wilk tests were not significant [for GC-RE, W(isj = 
0.94, p = 0.33; for GC-IR, W (m = 0.94, p = 0.30; for IC-RE, 
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FIGURE 2 | The raw facial temperature data of one guilty subject 
(subject 1) answering (A) 3 relevant questions and (B) 12 
irrelevant questions, and those of one innocent subject (subject 3) 



answering (C) 3 relevant questions and (D) 12 irrelevant 
questions, in which each line indicates the facial temperature for 
each question. 




FIGURE 3 | The smoothed facial temperature data under X = 10 of 
one guilty subject (subject 1) answering (A) 3 relevant questions 
and (B) 12 irrelevant questions, and those of one innocent subject 



(subject 3) answering (C) 3 relevant questions and (D) 12 irrelevant 
questions, in which each line indicates the facial temperature for 
each question. 
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FIGURE 4 | Scatter plots of the distributions of the facial temperature 
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W(i2) = 0.96, p = 0.84; for IC-IR W a2) = 0.96, p = 0.82], indi- 
cating that the normality assumption was not violated in any of 
the four conditions. 

GUILTY CONDITION 

A directional paired t-test for the GC revealed a significant differ- 
ence between the FTCVs for the RE questions (M = 0.40, SD = 
0.62) and the IR questions (M = 0.37, SD = 0.61), t (l7) = 1.91, 
p < 0.05. However, when utilizing the Bonferroni correction to 
control for an experimentwise error rate, the results were no 
longer significant and only showed a trend (p < 0.10) toward 
temperature responses to the crime relevant sub-questions being 
higher than the temperature responses to the crime irrelevant 
sub-questions. 

INNOCENT CONDITION 

A directional paired-samples t-test for the IC revealed no signif- 
icant difference between the FTCVs for the RE questions (M = 
-0.17, SD = 0.59) and the IR questions (M = —0.17, SD = 
0.59), t(n) = -0.04, p > 0.05 As expected, there were no dif- 
ferences in the temperature responses to crime relevant and 
irrelevant sub-questions. 

The above results show that there was no significant dif- 
ference in FTCV values between RE and IR responses in the 
IC, yet there was a noticeable trend in the values between RE 
and IR responses in the GC. These analyses were conducted 
using f-tests which analyze the data by comparing mean values. 
However, the data of the present study are time-based values, 
and a comparison of means may have been a meticulous enough 
approach. Important information may have been lost or over- 
looked during the process of averaging out this chronological 
data. Mean values summarize the data measured over contin- 
uous time points as mingle measures, and it may not, in this 
case, have been enough to consider only mean values to cap- 
ture all of the characteristics that reflect a group difference. 
Therefore, a functional ANOVA, which uses all of the values 
measured in its analysis, was utilized to evaluate all of the exist- 
ing data in its entirety in greater detail, as well as prevent any 



loss of information that may have occurred from a simple mean 
comparison. 

FUNCTIONAL ANOVA AND FUNCTIONAL DISCRIMINANT ANALYSIS 

The researchers who conducted the additional analyses did not 
participate in the actual experiment, and were only provided with 
the raw thermal data. This eliminated any researcher biases that 
may have affected the results. 

The functional ANOVA examined the main effect of con- 
dition (GC/IC), the main effect of relevance (RE/IR), and the 
interaction effect of condition and relevance. Figure 7A presents 
the mean facial temperature over the 10 s for the four differ- 
ent conditions. From top to bottom, the four lines indicate 
GC-RE, IC-IR, IC-RE, and GC-IR. We can see that guilty par- 
ticipants manifested higher facial temperature for RE questions 
than IR questions. Figure 7B shows the significant main effect 
of condition, which indicates that guilty participants mani- 
fested lower facial temperatures when answering IR questions 
over the 10 s by around 0.55°C. Figure 7C presents the signif- 
icant main effect of relevance, which indicates that innocent 
participants manifested lower facial temperatures when answer- 
ing RE questions compared to IR questions over the 10 s by 
around 0.33°C. Figure 7D shows that the interaction effect of 
condition and relevance was significant, which indicates that 
the facial temperature of guilty participants answering RE ques- 
tions was significantly higher than what could be predicted 
from the sum of the two main effects by 0.9° C. This suggests 
that facial temperature is affected by the interaction between 
condition (GC/IC) and relevance (RE/IR), meaning that guilty 
participants showed higher facial temperature when answering 
RE questions than IR questions whereas innocent participants 
did not. 

The functional discriminant analysis analyzed 90 functions 
(30 participants x 3 relevant questions) for the RE questions 
based on a weight function estimated with penalty parameter 
p = 10 determined by the leave-one-out cross-validation. Before 
the analysis, each function was baseline corrected by subtract- 
ing the corresponding baseline temperature. The 90 functions 
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of condition and relevance (solid line) with 95% pointwise confidence 
interval (dotted lines). 



were classified into two groups based on the weight function, and 
98.89% (89 out of 90) were correctly classified (misclassification 
rate = 1.11%). This result indicates that facial temperatures mea- 
sured while answering a RE question can be used to differentiate 
whether a participant is in the GC or 1C. 

For IR questions, 360 functions (30 participants x 12 irrel- 
evant questions) were analyzed based on a weight function 
estimated with penalty parameter p = 10 6 which was also deter- 
mined by the leave-one-out cross-validation. Again, before the 
analysis, each function was baseline corrected. When the 360 
functions were classified into two groups based on this weight 
function, 68.89% (248 out of 360) were correctly classified (mis- 
classification rate = 31.11%) which is only slightly higher than 
chance. This result indicates that facial temperatures measured for 
IR questions does not effectively discriminate guilty and innocent 
participants. 

DISCUSSION 

The present study utilized infrared thermal imaging with the 
CIT to detect the deception of participants who committed 
a mock crime. However, because there are certain limitations 
in using thermal imaging in the field, such as environmental 
factors and participant movement, the present study aimed to 
overcome these limitations by conducting a laboratory based 
experiment that would control for such variables. In accordance 
to conducting a lab based study, the present study further uti- 
lized deception detection techniques that were best suited for 



research purposes. One of which was to use the CIT method 
of questioning, which is based on empirical evidence, and the 
other being a highly realistic mock crime scenario. In addi- 
tion, a new and simple means of tracking the facial movement 
of the participants during thermal image measurement to min- 
imize temperature variances due to head movement was also 
developed. 

Results revealed that the average maximum skin surface tem- 
peratures recorded in the periorbital regions of the guilty partic- 
ipants were, as expected, significantly higher while responding to 
RE items compared to IR items. In contrast, and also as expected, 
there were no significant temperature differences between the RE 
and IR items measured from the innocent participants. These 
results are in line with the previous results of studies which 
used thermal imaging to detect deception (Pavlidis et al, 2002; 
Pollina et al, 2006; Tsiamyrtzis et al, 2007; Dowdall et al, 2009). 
However, the facial tracking process necessary to accurately mea- 
sure facial skin temperatures used in the present study was drasti- 
cally simplified in comparison to those used in previous research. 
Instead of relying on high-tech computer programming, a more 
analogue method of tracking was developed and was successfully 
applied. 

The results of this study support past research that the CIT is 
indeed an effective method of questioning for deception detec- 
tion, assuming the appropriate circumstances apply, which in 
this case was that the interviewer possessed information regard- 
ing evidence that only the guilty participants knew, and the 
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innocent participants did not. The CIT was conducted with no 
pre-interview or any other type of interviewee preparation, other 
than informing the subject that they would be asked a few ques- 
tions regarding a crime that had been committed. This allowed for 
an extremely short questioning session, the interviewer needed no 
information about the participant to conduct the session, no pre- 
interview or rapport building was necessary, and the interviewer 
needed no special training to conduct the questioning session. 
Therefore, when applicable, the CIT seems to be a much more 
efficient means of questioning than the CQT 

The mock crime used in the present study was also highly 
effective at making the participants feel as if they were actually 
committing a crime. How the participants felt during the mock 
crime was not systematically measured, yet it was clear to the 
experimenter that most of the participants were highly anxious 
about conducting the mock crime, as well as receiving the decep- 
tion detection procedure. Examples of this were, but not limited 
to, participants' hands shaking when they returned from conduct- 
ing the mock crime, participants not being able to steal the wallet 
and returning empty handed (but eventually going through with 
it), and in one extreme case the participant gave up and decided 
not to participate in the study after attempting the mock crime. 
The combination of the public location, having to dig through 
a stranger's bag for a wallet, and having to eliminate evidence at 
the computer lab assistant's computer seemed to have provided 
enough immersiveness to make the participants believe they were 
actually doing something illegal. 

In addition to the initial statistical analyses, additional analy- 
ses were conducted to further examine the results which revealed 
only a trend toward the predictions of the present study. As pre- 
dicted, and in line with the trend found in the initial analyses, the 
additional analyses conducted using a functional ANOVA were 
able to show that facial temperatures in the periorbital regions of 
guilty participants were significantly higher while responding to 
RE questions compared to IR questions, but not in the innocent 
participants. This result not only supports the results of previous 
studies, but also increased the ecological validity of the experi- 
ment by displaying consistent results even when analyzed through 
different statistical methods by researchers who did not partic- 
ipate in the experiment itself. A functional discriminant analysis 
was also able to discriminate between the guilty and innocent par- 
ticipants at a classification rate of 98.89%. This result provides 
support for the potential that thermal imaging has in detecting 
deception, or at the very least supplementing existing methods of 
deception detection to increase their accuracy. 

Certain limitations applied to the present study. First, the ther- 
mal imaging camera used was not the highest resolution camera 
available. There are other thermal imaging cameras currently 
available with greater resolution, which may produce more accu- 
rate measurements. Second, the participants were given a choice 
to choose between the GC and IC in order make the mock crime 
scenario more immersive. Although there were no significant dif- 
ferences in age, gender, health, or baseline temperatures between 
the two conditions, it is possible that other dissimilarities may 
have had an effect on the results, such as personality differences 
or intelligence. Had such information been measured prior to the 
condition selection procedure, it could have provided valuable 



information as to which participants chose the guilty condition 
and how they may have differed from the participants in the 
IC. Third, although the thermal imaging procedure was non- 
invasive compared to all the sensors of a polygraph, due to the 
fact that the participants were told not to move and maintain 
eye-contact with the camera during the questioning session, and 
that they had to have a small metallic sticker placed on their 
forehead, the procedure was not totally free of constraints. To 
overcome this limitation, the development of an advanced track- 
ing method will be necessary. Such a tracking method would 
allow for a more realistic study where the participants would 
be able to move freely during measurement. A more sophisti- 
cated tracking method could also prevent any changes in ratio 
between the AOI and the size of the participants' thermal image 
from moving back and forth in relation to the thermal imaging 
camera. Fourth, the study was conducted during the middle of 
summer, which may have led to less emphasized temperature dif- 
ferences between the innocent and guilty participants. In other 
words, the entire sample's baseline temperatures may have been 
higher than normal, leading to smaller increases in temperatures 
for the deceptive participants' facial temperature responses. Fifth, 
the number of participants in the study was relatively small. Even 
though the number was sufficient to conduct the statistical anal- 
yses without technical issues, future research should increase the 
number of participants to further increase the reliability of the 
results. Sixth, the participants were allowed to choose whether 
they wanted to engage in a mock-crime involving monetary risk, 
or a relatively risk-free task. The present study was conducted this 
way to further immerse the guilty participants into feeling as if 
they were really involved in the crime. Although the participants 
were random university students, allowing them to choose their 
own task forced the study to sacrifice a certain amount of control 
afforded by random allocation. However, in reality, most crim- 
inals decide for themselves whether they should or should not 
commit criminal behavior, and therefore this freedom of choice 
may have increased ecological validity. A final limitation is that 
the present study used the value of the hottest single pixel of each 
frame from within the AOI for the analyses and from a statistical 
point of view this is not a very robust approach. 

The results of the present study have demonstrated three main 
findings. First, it has provided support for previous studies that 
have utilized thermal imaging to detect deception, but in an 
experimental environment further controlling for temperature, 
humidity, and unnecessary body movement, in a much more sim- 
ple and effective manner. Second, it has provided support for 
previous studies that claim the CIT is a more efficient questioning 
method requiring little to no training. Finally, a mock crime was 
designed that seems highly effective at providing a realistic crime 
experience, without placing anyone involved at risk or danger. 

In conclusion, the present study has shown that using thermal 
imaging to detect deception has realistic and applicable poten- 
tial to be utilized in modern day law enforcement. However, 
standardization of the equipment, methodology, and data analy- 
sis techniques are necessary before any kind of field application 
can be expected. Future research on deception detection using 
thermal imaging should place emphasis on three areas. First, 
developing a more advanced facial tracking method. Second, a 
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simpler way of analyzing the thermal data collected, to make 
detecting faster and more accurate, yet easier to apply to real- 
world circumstances. Third, conducting research using the most 
high-resolution thermal imaging equipment. This may produce 
not only more accurate results, but even allow for the discovery 
of previously unknown physiological changes in facial skin tem- 
peratures or facial temperature changing regions that can also act 
as cues during deception detection. Finally, taking a more robust 



approach in the statistical analyses of the maximum temperature 
values by analyzing not one pixel, but an area of pixels from the 
thermal images. 
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APPENDIX 

ENVELOPE SELECTION EXPLANATION 

"This experiment involves you conducting a mission. There are 
two different missions you will be choosing from. Both missions 
will be conducted at the same location. It is a public place with 
people who are not involved with the experiment. There may only 
be a small number of people, or quite a few. 

In my hands are two different envelopes. The blue envelope 
contains a legal mission and the red envelope contains an illegal 
mission. If you choose the blue mission, you will simply do what 
the mission says, and as long as you correctly do what the mission 
states, you will receive $10. However, if you choose the red mis- 
sion, you will only receive a reward if you successfully complete 
the two the stages of the mission. 

To complete the first stage, you will have to complete the stated 
illegal act without anyone else knowing other than myself. If any- 
one else discovers that you are committing an illegal act, you 
automatically fail the mission. After successfully completing the 
mission, you will be questioned while having your bio-signals 
measured. 

To complete the second stage of the mission, you must fool 
the questioner into thinking you did not commit any illegal 
acts, without being detected. This means that you must not only 
convince the questioner, but you must also trick the bio-signal 
measurement equipment as well. So, if you successfully complete 
both stages of the red mission, you will be rewarded $10 for par- 
ticipating in the experiment, as well as a bonus reward of $20, for 
a total of $30. However, if you fail either stage of the red mission, 
you will receive nothing, and go home empty handed. 

If you have any questions for me, please feel free to ask. If not, 
select an envelope now." 

SMOOTHING: ROUGHNESS PENALTY SMOOTHING METHOD 

Let y t denote the value of facial temperature at time t. In func- 
tional data analysis, the observed temperature y t , is regarded 
as a realization of an underlying continuous smooth temper- 
ature function x, rather than merely as a sequence of discrete 
observations. More specifically, we assume the following model, 



y t = x(t) + s(0 



(Al) 



where x(t) is the value of the underlying smooth function evalu- 
ated at time t and g(f) is a perturbation at time t that causes the 
observed data y t to look rough. The first step of functional data 
analysis is to estimate the smooth function x, which requires a 
smoothing method to convert the observed values y t to a function 
x with values x(t) computable for any desired time point t. 

Smoothing methods based on so-called basis expansion pro- 
cedures represent a function x as a weighted sum of well-known 
basis functions $ k 



(A2) 



of well-known basis functions, which aids further analysis. In 
particular, it is convenient to estimate derivatives of a function 
expressed by a basis expansion because the first derivative of x, 
Dx, can be expressed as 



Dx(s) = CkD$k(s), 



(A3) 



where D$k(s) is the first derivative of basis function <\> k at time s. 
More generally, the derivative of order m of function x at time s 
will be given as 

K 

D m x{s) = J2 c k D m $ k (s), (A4) 
k= 1 

where D m denotes the derivative of order m. This property will be 
useful in estimating the coefficients c k , which will be clear soon. 

There are many popular bases that are widely used in prac- 
tice. Most functional data analyses are known to involve either a 
Fourier basis for periodic data or a B-spline basis for non-periodic 
data (Ramsay and Silverman, 2006). We used the B-spline basis 
(de Boor, 2001) for smoothing the data because facial tempera- 
ture can be considered non-periodic. 

The remaining problem is how to estimate the coefficients c k - 
We applied a penalized least-squares method that is consid- 
ered more powerful and versatile than other methods such as 
least-squares smoothing, kernel smoothing, and local polynomial 
fitting approaches (Ramsay and Silverman, 2006). The objective 
function of the penalized least-squares method is given as the 
following. 



T 

t= i 



k= 1 



/ [D 2 x(s)Y 



ds 



(A5) 



Basically, the coefficients are obtained by minimizing the sum of 
squared differences between observed values y, and function val- 
ues x as shown in (A6), the first term of the objective function. 



E 



-,2 



c k <bk(t) 



k=l 



(A6) 



The role of the second term in the objective function is to control 
the roughness of the function x. The squared second derivative 
[D 2 x(s)] 2 of function x at time 5 is called its curvature at s, since 
a straight line, which has no curvature, will have a zero second 
derivative. Therefore, a function's roughness measured across all 
time points s is the integrated squared second derivative as given 
in (A7). 

(A7) 



j [D 2 x(s)f ds 



By applying (4) this roughness can be rewritten as 

-, 2 



k= 1 



/ 



K 



J2 c k D 2 4> k (s) 



.k= 1 



ds 



(A8) 



where K is the number of basis functions and c k is the coefficient 
for the fcth basis function. By estimating the coefficients c k , we can 
represent a complicated-looking function as a linear combination 



which shows that the basis expansion is useful in estimating coef- 
ficients c k ; it expresses the roughness as a linear combination of 
the second derivative of well-known basis functions. 
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By minimizing the objective function (A5), we wish to obtain 
two conflicting goals in curve estimation: model fit and general- 
izability. By minimizing (A6), we want to obtain the estimated 
curve with a good fit to the data in terms of minimizing the 
residual sum of squares. On the other hand, by minimizing 
(A7), we do not want the fit to be too good as to be exces- 
sively wiggly and overfit the data. The balance between these 
two conflicting goals in the objective function can be controlled 
by the smoothing parameter X. As X gets bigger, the objec- 
tive function will place more emphasis on the smoothness and 
less on fitting the data. Therefore, as X approaches infinity, the 
estimated curve will approach the standard linear regression 
which has J[D 2 x(s)] 2 ds = 0. In contrast, as X becomes smaller, 
less penalty is placed on the curvature, and as X approaches 
zero, the estimated curve approaches an interpolant to the data, 
which passes exactly though all the given data points. There 
exist several methods to choose the smoothing parameter and 
we applied the generalized cross-validation (GCV) method pro- 
posed by Craven and Wahba (1979). The basic idea under GCV 
is to compute a measure of mean squared error over a range 
of values of X and choose the value that gives its minimum. 
In this analysis, we tried 1 1 different values of X (log 10 X = 
-5,-4,-3,-2, -1, 0, 1, 2, 3, 4, 5) and obtained X = 10 1 
as the optimal value. 

FUNCTIONAL ANOVA 

After obtaining a smooth function x, we can perform a functional 
ANOVA. The form of the functional ANOVA model for analyzing 
our data can be given as 

Temp,(0 = u,(f) + a(t) + P(t) + y(t) + 8,(0 + e,(t) (A9) 

where, Temp ; (f) is the facial temperature for participant i eval- 
uated at time f, |x(t) is the mean facial temperature of inno- 
cent participants answering irrelevant questions evaluated at 
time t, a(t) is the main effect of group (i.e., the difference 
between the mean face temperature of innocent participants 
and that of guilty participants when they are answering irrel- 
evant questions), P(f) is the main effect of relevance (i.e., the 
difference between the mean face temperature of innocent par- 
ticipants answering irrelevant questions and that of innocent 
participants answering relevant questions), y(f) is the interac- 
tion between group and relevance (i.e., the difference between 
the mean face temperature of guilty participants answering irrel- 
evant questions and that of guilty participants answering relevant 
questions), 8;(f) is the participant specific effect, and e,(f) is a 
residual function. 

With this model, we want to test whether there is a signif- 
icant effect of being in the guilty group, being in the relevant 
condition, and/or simultaneously being in the guilty group and 
relevant condition, on facial temperatures measured over time. In 
order to estimate parameters (or functions), we need to construct 
a matrix Z where N is the total number of functions {N = 150, 
30 participants x 15 questions). The rows corresponding to the 
participants in group 1 (innocent) and condition 1 (irrelevant) 
will have [1 0 0 0], the rows corresponding to the participants 
group 2 (guilty) and condition 1 (irrelevant) will have [1 1 0 0], 



the rows corresponding to the participants in group 1 (innocent) 
and condition 2 (relevant) will have [10 10], and the rows corre- 
sponding to the participants in group 2 (guilty) and condition 2 
(relevant) will have [1 1 1 1] in the first four columns. In the next 
18 columns, subject k in group 1 (guilty) will have a row vector 
whose kth element is one and all the other elements are zero. In 
the next 12 columns, subject k in group 2 (innocent) will have a 
row vector whose kth element is one and all the other elements 
are zero. Then the model (A9) has the equivalent formulation as 
linear regression as shown in (A10), 

x(f) = ZP(f) + e(f) (A10) 

where x(f) is N by 1 vector of facial temperature, P(f) = 
[|x(f), a(f), P(t), Y(f), Si(f). . . . .8jv(f)]', and e(f) is N by 1 vec- 
tor of residuals. In order to deal with linear dependency in the 
columns of the design matrix, we need to augment the rows of 
the design matrix to enforce the sum of the participant specific 
effects in each group to be zero. 

The regression coefficients (3(f), are the parameters that we 
want to estimate and they can be estimated by minimizing the 
following objective function. 

Lp = J (Ms) - ZP(s)]'[x(s) - ZP(s)])ds 

+ Xp J ([D 2 p(s)]'[D 2 P(s)])ds (All) 

This objective function has the same structure as the objective 
function (A5) of the penalized least-squares smoothing. Basically, 
the regression coefficients are obtained by minimizing sum of 
squared differences between the function value x(s) and the pre- 
dicted value ZP(s) over all possible values of s, i.e., minimizing 
(A12), the first term of the objective function. 

j ([x(s) - Zp(s)]'[x(s) - ZP(s)])<fc (A12) 

The second term of the objective function is introduced to con- 
trol the roughness of the regression coefficient function (3. The 
inner product of the second derivative of P(s), [D 2 P(s)]'[D 2 P(5)], 
is defined as the curvature of the function ft at time s. Therefore, 
the function's roughness measured across all argument values s is 
the integrated curvature as given in (A13). 

j ([D 2 P(5)]'[D 2 p(5)])rfs (A13) 

As P can be considered a smooth function, we used a basis expan- 
sion method to represent (3. Let B denote a 2 by -JCp matrix of 
coefficients, where is the number of basis functions, and 0(5) 
denote a Kp by 1 vector of basis functions at time s. We used the B- 
spline basis for smoothing the function of regression coefficients. 
The regression coefficients can now be represented as (A14) by 
the basis expansion. 

POO = B9(s) (A14) 



Frontiers in Human Neuroscience 



www.frontiersin.org 



March 2013 | Volume 7 | Article 70 | 13 



Park et al 



Functional analysis of thermographic lie-detection 



Therefore, the objective function (All) can be rewritten as 
(A15) and estimating regression coefficients P is equivalent to 
estimating the coefficients of the basis expansion B. 

Lp = J ([ X (s) - ZB8(s)]'[x(s) - ZB9(s)])ds 

+ Xp j ([BD 2 Q(s)]'[BD 2 Q(s)])ds (A15) 

The coefficient B, can be estimated by minimiz- 
ing Lp with respect to B, which is given as vec(B) = 

[Jee(g>(Z'Z) + R®Xpl]~ 1 vec(Z'CJ < | ) 9), where vec(B) is an 
operator that stacks the columns of the matrix B in one col- 
umn vector, Cg) is the Kronecker product, C is the matrix of 
coefficients in (A2), Jee = fd(s)Q(s)'ds, J^e = f §(s)Q(s)'ds, and 
R = fD 2 d(s)D 2 d(s)'ds. 

The smoothing parameter Xp again plays a role to control 
the level of the roughness of the estimated function of P(s). As 
Xp gets bigger, the estimated curve of ft(s) will be smoother. In 
order to choose the value of Xp, we applied a cross-validation 
method proposed by Ramsay et al. (2009). The basic idea of this 
method is to compute the cross validated integrated squared error 
over a range of values of Xp and choose the value that gives its 
minimum. In this analysis, we tried 13 different values of Xp 
(log 10 Xp = 0, 1, 2, 3, 4, 5, 6, 7, 8, 9) and obtained Xp = 10 6 as the 
optimal value. 

FISHER'S LINEAR DISCRIMINANT ANALYSIS 

The purpose of Fisher's linear discriminant analysis is to seek 
components (or linear combinations) of measured variables that 
are efficient for discrimination. Suppose that we have a set of N 
observations (or objects) of p dimensions denoted by xi , . . . , xjy, 
which consists of C subsets, N\ in the subset Si, N2 in the subset 
S2, ■ ■ ■, Nc in the subset Sc- If we form a linear combination of 
the p variables in x,-, it can be represented by 



w'x; 



(Al) 



where w indicates a weight vector of order p and y, indicates the 
constructed component of observation i. If the norm of w is one, 
constructing a component by w'x,- is equivalent to projecting x; 
onto a line in the direction of w geometrically. We want to find 
the best direction w that separates C subsets as much as possible. 
The measure of the separation between the projected data y can 
be measured by the variance of the means of the C subsets. If x is 
the grand mean of x, given by 



(A2) 



>'= 1 



then the grand mean for the corresponding projected data is 
given by 

1 N 1 N I 1 N \ 

i=l i=I \ i=l / 



which is the projected grand mean. Likewise, if x c is the sample 
mean of x in subset c given by 



x c = — x; 



(A4) 



then the sample mean for the corresponding ys in subset c is 
given by 

y c = — y, = — w'x, = w' I — x I = v/xr. 
' N c ^ ' N c N ^ J 

C yi € S c C x; € S c \ Xj € S c j 

(A5) 

which is the projected subset mean. The variation among the 
projected means can be calculated by 



4 = J^Ncifc-y) 2 

c= 1 
C 

c= 1 

C 

= ^Ny(x c -i) (x c -x)'w 



c= 1 



C 



^]N c (x c -x)(x c -x)' 



_c= 1 

w'Sbw 



(A6) 



where S# = Y^c= 1 N c (x c — x) (x c — x)'. We can make this vari- 
ation as large as we wish by multiplying a constant to w. To 
obtain good separation, this variance should be large relative to 
the variation within each subset, which can be measured by 

C 

s w = J2J2 b>i-yc) 2 

c=l yi €S c 

= EE(^-A) 2 

C= lXj€S c 

c 

= ^ ^ W 7 (X, - Xc) (X, - Xc)' W 
c= lx,€S c 



w 



c=lx,eS c 



(A7) 



where Sw = J2^= 1 H XieS c ( x i — x c ) (x, — x c )'. Fisher's linear dis- 
criminant analysis seeks to find w that maximizes the following 
objective function 



J(w) = -f- 



W Sbvt 



Sur w'SwW 



(A8) 
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Maximizing (A8) with respect to w is equivalent to maximiz- 
ing w'Sbw subject to the constraint, vJ Sww = 1, which can be 
obtained by maximizing the following objective function with 
respect to w and a Lagrange multiplier X 



/(w, X) = w' S#w — X (W Siyw — l) . 



(A9) 



Taking a derivative of (A9) with respect to w and setting it to zero 
yields 

S B w = XSww (A10) 

which is a generalized eigenvalue problem. Taking a derivative of 
(A9) with respect to X and setting it to zero yields the constraint 



w'Sww = 1. 



(All) 



If we premultiply (A10) by W on both sides, we can obtain the 
value of X 

W Srw 

w'S]jw (A12) 



X 



w'Svyw 



which is the maximum variance among the projected means. 
Therefore, the eigenvalue of the eigen-equation (A10) indicates 
the maximum variance among the projected means that can be 
achieved and the eigenvector indicates the direction, or weight, 
that maximally separates C subgroups. 

If one dimension is not enough to separate C subgroups, 
we can consider more eigenvalue-eigenvector pairs that can be 
obtained by solving the generalized eigenvalue problem given by 
(A10). If there are C subgroups, C — 1 eigenvalue-eigenvector 
pairs are usually used to separate C subgroups. This is equiv- 
alent to constructing C — 1 linear combinations of variables 
given by 

Yi = Wx, (A13) 

where W = [wi , . . . , wc- 1 ] ' and y is the vector of C — 1 compo- 
nents. 

After obtaining W, we can allocate a new observation x new into 
one of C subgroups in the following way. First we calculate y nevl = 
Wx new and calculate the distance between y new and the projected 
mean of each subgroup given by 

dist c = (y- w - yc )V ew -yc) 

= (Wx 11 ™ - Wx c )' (Wx 11 ™ - Wx,) 

= (x new - x c )' WW (x new - x c ) . (A14) 

Then the new observation is allocated to the subgroup that yields 
the smallest distance value. 

FUNCTIONAL DISCRIMINANT ANALYSIS 

Fisher's linear discriminant analysis can be readily extended to 
functional data. Suppose we have a set of N functions measured 
at T time points, denoted by X{{t), where i = 1, . . . , N and t = 
1, . . . , T, and those N observations belong to C subsets with N\ in 
the subset Si , N2 in the subset S2, ■ ■ ., Nc in the subset Sc- Fisher's 
linear discriminant analysis aims to find linear combinations of 



variables that separate C subsets as much as possible. For func- 
tional data, finding a linear combination of observed variables 
corresponds to finding a weighted integration of the functions 
given by 



y, = j %(t)x,(t)dt 



(A15) 



where ^(f) indicates the weight function evaluated at time f andy; 
is the component constructed by the weighted integration of the 
observed function x, (f). 

In order to estimate the weight function ^(f), we will adopt a 
basis function expansion approach to approximate functions. Any 
function can be approximated up to some degree of approxima- 
tion by a linear combination of suitable basis functions. Suppose 
that 4>(t)is an M by 1 vector of M suitable basis functions evalu- 
ated at time t. Then the observed functions jc,(f), and the weight 
function %(t), can be represented by basis function expansions as 
follows 



x,(t) = v>(f) = cKO'v; 



i=(f) = W<K0 = <Kt)'w 



(A16) 



(A17) 



where v; is the M by 1 vector of the coefficients of basis functions 
for x;(f) and w is the M by 1 vector of the coefficients of basis 
functions for %(t). Then the weighted integration (A15) can be 
rewritten as 



<Kt)4>(t)'& 1 ' 



w' 



w'jv, 



(A18) 



where J = f <$>(t)ty(t)' dt is the M by M symmetric matrix of inner 
products of the basis functions. 

We want to find the weight function %(f), or equivalently, the 
coefficients of its basis functions w, that separate C subsets as 
much as possible. The measure of the separation between the pro- 
jected datay can be measured by the variance of the means of the 
C subsets. The grand mean for the corresponding component is 
given by 



l N j N / 1 N \ 

y = n & = n £ w ' lv ' = W ' J (n 2>) 

1=1 1=1 \ 1=1 / 



w'Tv. (A19) 



where v is the mean vector of the coefficients of basis functions 
for Xi(t). Likewise, the sample mean for the y,'s in subset c is 
given by 



yc = w E n = ^ E w '' v < = w 'J ^ E v > = w '^- 

c yi €S c c v,€5 c \ v;eS c / 

(A20) 

where \ c is the mean vector of the coefficients of basis functions 
for Xj(t) in subset c. The variation among the projected means can 
be calculated by 



Frontiers in Human Neuroscience 



www.frontiersin.org 



March 2013 | Volume 7 | Article 70 | 15 



Park et al 



Functional analysis of thermographic lie-detection 



c=\ 

C 

^N c (w'Jv c -w / Jv) 2 

c= 1 

C 

^NXl(v c -v)(v c -v)'jw 



w'J 



N c - V) (V C - V)' 



Jw 



w'JSbJw 



(A21) 



where Sb = X ^c(v c — v)(v c — v)'. We want this variation as 

c= 1 

large as possible relative to the variation within each subset, which 
can be measured by 

c 

c= lyi€S c 

= EE (w'Jv.-w'jv,) 2 

c=lv i eS c 

C 

c=lv;€S c 



c =l Vj eS c 



Jw 



= w'JSwJw 



(A22) 



where Sw = X! X! (v, — v c )(v,- — v c )'. Fisher's linear discrim- 

c = 1 Vj € S c 

inant analysis seeks to find w that maximizes the following 
objective function 



4 w'JSbJw 
/( W ) = -f = . 

w'JSwJw 



(A23) 



'Ml' 



If we use enough number of basis functions, for example, the 
same number as the number of time points, the estimated weight 
function ^(r) = w / rj)(f) could be jagged, which would make the 
interpretation of the weight function difficult. We would prefer a 
smoother version of the weight function, which can be obtained 
by maximizing the following regularized objective function 



7(w, p) = 



w'JSbJw 



w'JS w Jw+p/(D^(0) z dt 
w'JSbJw 

w'JS w Jw+ p fD 2 Wd\>(t)D 2 (\>(ty-wdt 



w'JSbJw 



WJSwJw + pw ' (/ D 2 fy(t)D 2 §(tydt) w 
w'JSbJw 



w'JSwJw + pw'Rw 



(A24) 



where D 2 indicates the second derivative of a function and 
R= / D 2 §(t)D 2 §{t)'dt. The penalty term added to the objec- 
tive function w'Rw is the squared integration of the second 
derivative of the weight function, which measures the roughness 
or smoothness of the weight function. The penalty parameter 
p(>0) controls the importance of the penalty term. A larger 
value of penalty term will put more emphasis of the smooth- 
ness of the weight function and yield a smoother weight function. 
The optimal value of p can be obtained by a cross-validation 
method. 

Maximizing (A24) with respect to w is equivalent to maxi- 
mizing w'JSbJw subject to the constraint w'JSwJw + pw'Rw = 1, 
which can be obtained by maximizing the following objective 
function with respect to w and a Lagrange multiplier X 

/(w, X) = w'JSbJw - X (w'JS w Jw + pw'Rw) . (A25) 

Taking a derivative of (A25) with respect to w and setting it to 
zero yields 



JSbJw = X(JS W J + pR)w 



(A26) 



which is a generalized eigenvalue problem. Taking a derivative of 
(A25) with respect to X and setting it to zero yields the constraint 



w / JSwJw-|- pw'Rw = 1. 



(A27) 



If we premultiply (A26) by W on both sides, we can obtain the 
value of X 



w'JSbJw 



w'JSvtJw-l- pw'Rw 



w'JSbJw 



(A28) 



which is the maximum variance among the means of the com- 
ponent y,. Therefore, the eigenvalue of the eigen-equation (A26) 
indicates the maximum variance among the means of the com- 
ponent that can be achieved and the eigenvector indicates the 
coefficients of the weight function that maximally separate C 
subgroups. 

If one dimension is not enough to separate C subgroups, 
we can consider more eigenvalue-eigenvector pairs that can be 
obtained by solving the generalized eigenvalue problem given by 
(A10). If there are C subgroups, C — 1 eigenvalue-eigenvector 
pairs are usually used to separate C subgroups. This is equivalent 
to constructing C — 1 linear combinations of variables given by 



WJv, 



(A29) 



where W = [wi, . . . , wc - i]' and y is the vector of C — 1 
components. 

After obtaining W, we can allocate a new observation 
x^™(t) = <\>(t)'v n ™ into one of the C subgroups in the following 
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way. First we calculate y new = WJv new and calculate the distance 
between y"™ and the projected mean of each subgroup given by 

dist c =(r evf -fc)'{Y Dew -Yc) 

= (WJv new - WTv c )' (WJ/™ - WJv c ) 

= (V 1 ™ - v c )' W'JJW (V 1 ™ - v c ). (A30) 

Then the new observation is allocated to the subgroup that yields 
the smallest distance value. 

CONCEALED INFORMATION TEST QUESTIONING PROTOCOL 

Primary question 1: "What was the item you stole from the 
computer lab?" 

Sub-question 1-1: "Was it a ring?" 

Sub-question 1-2: "Was it a wallet?" 



Sub-question 1-3: "Was it a pair of glasses?" 

Sub-question 1-4: "Was it a notepad?" 

Sub-question 1-5: "Was it a jewel?" 
Primary question 2: "What did you do before you left the 
computer lab?" 

Sub-question 2-1: "Did you clean the room?" 

Sub-question 2-2: "Did you meet a friend?" 

Sub-question 2-3: "Did you take a picture?" 

Sub-question 2-4: "Did you turn off the webcam?" 

Sub-question 2-5: "Did you have a drink?" 
Primary question 3: "What was the item you stole from the 
computer lab?" 

Sub-question 3-1: "Was it a jewel?" 

Sub-question 3-2: "Was it a notepad?" 

Sub-question 3-3: "Was it a watch?" 

Sub-question 3-4: "Was it a wallet?" 

Sub-question 3-5: "Was it a pair of glasses?" 
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